{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Data Preparation and Feature Engineering\n",
        "\n",
        "Data is critical to training and preparing a model. In this notebook we will cover how to load data into ML.NET and ensure it is in the proper format so that ML.NET can work with it. \n",
        "\n",
        "In this notebook you will learn how to... \n",
        "\n",
        "- Load data into ML.NET\n",
        "- Apply data transforms to help ML.NET understand the data"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Loading data in ML.NET"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### What is an IDataView?\n",
        "\n",
        "The [IDataView](https://docs.microsoft.com/dotnet/api/microsoft.ml.idataview?view=ml-dotnet) is the data format ML.NET loads for training. It is a set of interfaces and components that provide efficient, compositional processing of schematized data for machine learning and advanced analytics applications. It is designed to gracefully and efficiently handle high dimensional data and large data sets. \n",
        "\n",
        "The IDataView has general schema support, in that a view can have an arbitrary number of columns, each having an associated name, index, data type, and optional annotation."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### How to create an IDataView\n",
        "\n",
        "You can create an IDataView by using any of the methods for loading data:\n",
        "\n",
        "- TextLoader\n",
        "- LoadFromEnumerable\n",
        "- DatabaseLoader\n",
        "- LoadFromTextFile\n",
        "\n",
        "See [Load Data from files and other sources](https://docs.microsoft.com/dotnet/machine-learning/how-to-guides/load-data-ml-net) for further documentation and examples. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "dotnet_interactive": {
          "language": "csharp"
        },
        "vscode": {
          "languageId": "csharp"
        }
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div><div></div><div></div><div><strong>Installed Packages</strong><ul><li><span>Microsoft.ML, 2.0.0-preview.22313.1</span></li></ul></div></div>"
            ]
          },
          "execution_count": 1,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "#i \"nuget:https://pkgs.dev.azure.com/dnceng/public/_packaging/MachineLearning/nuget/v3/index.json\"\n",
        "#r \"nuget: Microsoft.ML, 2.0.0-preview.22356.1\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "dotnet_interactive": {
          "language": "csharp"
        },
        "vscode": {
          "languageId": "csharp"
        }
      },
      "outputs": [],
      "source": [
        "using Microsoft.ML;\n",
        "using Microsoft.ML.Data;\n",
        "using Microsoft.ML.Transforms; "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Download or Locate Data\n",
        "The following code tries to locate the data file in a few known locations or it will download it from the known GitHub location."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "dotnet_interactive": {
          "language": "csharp"
        },
        "vscode": {
          "languageId": "csharp"
        }
      },
      "outputs": [],
      "source": [
        "using System;\n",
        "using System.IO;\n",
        "using System.Net;\n",
        "\n",
        "string EnsureDataSetDownloaded(string fileName)\n",
        "{\n",
        "\n",
        "\t// This is the path if the repo has been checked out.\n",
        "\tvar filePath = Path.Combine(Directory.GetCurrentDirectory(),\"data\", fileName);\n",
        "\n",
        "\tif (!File.Exists(filePath))\n",
        "\t{\n",
        "\t\t// This is the path if the file has already been downloaded.\n",
        "\t\tfilePath = Path.Combine(Directory.GetCurrentDirectory(), fileName);\n",
        "\t}\n",
        "\n",
        "\tif (!File.Exists(filePath))\n",
        "\t{\n",
        "\t\tusing (var client = new WebClient())\n",
        "\t\t{\n",
        "\t\t\tclient.DownloadFile($\"https://raw.githubusercontent.com/dotnet/csharp-notebooks/main/machine-learning/data/{fileName}\", filePath);\n",
        "\t\t}\n",
        "\t\tConsole.WriteLine($\"Downloaded {fileName}  to : {filePath}\");\n",
        "\t}\n",
        "\telse\n",
        "\t{\n",
        "\t\tConsole.WriteLine($\"{fileName} found here: {filePath}\");\n",
        "\t}\n",
        "\n",
        "\treturn filePath;\n",
        "}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### Loading from a file\n",
        "\n",
        "A [TextLoader](https://docs.microsoft.com/dotnet/api/microsoft.ml.data.textloader?view=ml-dotnet) can load a structured file into an IDataView. Structured information is represented as columns and rows of data. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "The IDataView has general schema support, in that a view can have an arbitrary number of columns, each having an associated name, index, data type, and optional annotation. You can define the schema for your data using Plain-Old-CLR-Objects (POCO) or classes.\n",
        "\n",
        "A few things to notice about the ModelInput class.\n",
        "- The `LoadColumn` attribute specifies the column indices. This is a necessary attribute when loading from a file. \n",
        "- The `ColumnName` attribute used to set the name of the column to something other than the property name. The in-memory objects use the property name. However, for data processing and building machine learning models, ML.NET overrides and references the property with the value provided in the ColumnName attribute."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "dotnet_interactive": {
          "language": "csharp"
        },
        "vscode": {
          "languageId": "csharp"
        }
      },
      "outputs": [],
      "source": [
        "public class ModelInput\n",
        "{\n",
        "    [LoadColumn(0)]\n",
        "    [ColumnName(@\"vendor_id\")]\n",
        "    public string Vendor_id { get; set; }\n",
        "\n",
        "    [LoadColumn(1)]\n",
        "    [ColumnName(@\"rate_code\")]\n",
        "    public float Rate_code { get; set; }\n",
        "\n",
        "    [LoadColumn(2)]\n",
        "    [ColumnName(@\"passenger_count\")]\n",
        "    public float Passenger_count { get; set; }\n",
        "\n",
        "    [LoadColumn(3)]\n",
        "    [ColumnName(@\"trip_time_in_secs\")]\n",
        "    public float Trip_time_in_secs { get; set; }\n",
        "\n",
        "    [LoadColumn(4)]\n",
        "    [ColumnName(@\"trip_distance\")]\n",
        "    public float Trip_distance { get; set; }\n",
        "\n",
        "    [LoadColumn(5)]\n",
        "    [ColumnName(@\"payment_type\")]\n",
        "    public string Payment_type { get; set; }\n",
        "\n",
        "    [LoadColumn(6)]\n",
        "    [ColumnName(@\"fare_amount\")]\n",
        "    public float Fare_amount { get; set; }\n",
        "\n",
        "}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "All ML.NET operations start in the [MLContext](https://docs.microsoft.com/dotnet/api/microsoft.ml.mlcontext) class. Initializing mlContext creates a new ML.NET environment that can be shared across the model creation workflow objects. It's similar, conceptually, to DBContext in Entity Framework."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "dotnet_interactive": {
          "language": "csharp"
        },
        "vscode": {
          "languageId": "csharp"
        }
      },
      "outputs": [],
      "source": [
        "//Create MLContext\n",
        "MLContext mlContext = new MLContext();"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Create a [TextLoader](https://docs.microsoft.com/dotnet/api/microsoft.ml.data.textloader?view=ml-dotnet) based on the ModelInput type. Then use the text loader to load from the data file. At minimum, the loader will need to be told if the file has a header, and what separator character the file uses. \n",
        "\n",
        "You could also use the direct method [LoadFromTextFile](https://docs.microsoft.com/dotnet/api/microsoft.ml.textloadersavercatalog.loadfromtextfile?view=ml-dotnet). The advantage of the TextLoader is that it gives the option to load files from multiple files from different locations. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "dotnet_interactive": {
          "language": "csharp"
        },
        "vscode": {
          "languageId": "csharp"
        }
      },
      "outputs": [],
      "source": [
        "var trainDataPath = EnsureDataSetDownloaded(\"taxi-fare.csv\");\n",
        "\n",
        "// Create TextLoader based on the Model Input type. \n",
        "TextLoader textLoader = mlContext.Data.CreateTextLoader<ModelInput>(separatorChar: ',', hasHeader: true);\n",
        "\n",
        "// Load the data into an IDataView. Load() method can support multiple files. \n",
        "// Files must they have the same separator character, header, column names, etc. \n",
        "IDataView data = textLoader.Load(trainDataPath);\n",
        "\n",
        "data.Preview(1); "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### Loading in memory collection\n",
        "\n",
        "ML.NET supports loading data from an in memory collection. This makes it easy to load from a JSON or XML file using C#. Learn how to [deserialize JSON with C#](https://docs.microsoft.com/dotnet/standard/serialization/system-text-json-how-to?pivots=dotnet-6-0#how-to-read-json-as-net-objects-deserialize) or use [XML serializer](https://docs.microsoft.com/dotnet/api/system.xml.serialization?view=net-6.0) to get those files into memory. \n",
        "\n",
        "Once you have the data collection in memory, you can load it into ML.NET with the [LoadFromEnumerable](https://docs.microsoft.com/dotnet/api/microsoft.ml.dataoperationscatalog.loadfromenumerable?view=ml-dotnet) method. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "dotnet_interactive": {
          "language": "csharp"
        },
        "vscode": {
          "languageId": "csharp"
        }
      },
      "outputs": [],
      "source": [
        "ModelInput[] inMemoryCollection = new ModelInput[]\n",
        "{\n",
        "    new ModelInput\n",
        "    {\n",
        "        Vendor_id = \"CMT\",\n",
        "        Rate_code = 1,\n",
        "        Passenger_count = 1,\n",
        "        Trip_time_in_secs = 1271,\n",
        "        Trip_distance = 3.8f,\n",
        "        Payment_type = \"CRD\",\n",
        "        Fare_amount = 17.5f,\n",
        "    },\n",
        "    new ModelInput\n",
        "    {\n",
        "        Vendor_id = \"VST\",\n",
        "        // missing Rate_code\n",
        "        Passenger_count = 1,\n",
        "        Trip_time_in_secs = 474,\n",
        "        Trip_distance = 1.5f,\n",
        "        Payment_type = \"CSH\",\n",
        "        Fare_amount = 8, \n",
        "    }\n",
        "};"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "dotnet_interactive": {
          "language": "csharp"
        },
        "vscode": {
          "languageId": "csharp"
        }
      },
      "outputs": [],
      "source": [
        "// Create MLContext\n",
        "MLContext mlContext = new MLContext();\n",
        "\n",
        "//Load Data\n",
        "IDataView data = mlContext.Data.LoadFromEnumerable<ModelInput>(inMemoryCollection);\n",
        "\n",
        "data.Preview(1);"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### What's the difference between a DataFrame and IDataView?\n",
        "\n",
        "You may have heard of the [DataFrame](https://docs.microsoft.com/dotnet/api/microsoft.data.analysis.dataframe?view=ml-dotnet-preview) type. It is another tool to load, view and manipulate data that is common to Notebooks. It implements an IDataView, so it can easily be passed to ML.NET.\n",
        "\n",
        "DataFrame and IDataView are very similar in the sense that they both are ways of representing data in a tabular format and applying transformations for it. Some key differences:\n",
        "\n",
        "- DataFrame only supports loading delimited files.\n",
        "- DataFrame runs on memory so you're limited to the amount of memory on your PC.\n",
        "\n",
        "The DataFrame is recommended when performing tasks like exploratory data analysis on a sample of your data. Look at the reference notebook [REF-Data Processing](https://github.com/dotnet/csharp-notebooks/blob/main/machine-learning/REF-Data%20Processing.ipynb) for an example of using Data Frames to manipulate a data file for training.\n",
        "\n",
        "IDataView is recommended for training on larger datasets, and what is used for the examples in this notebook. Larger datasets in this case are defined as datasets that can't fit into memory."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Data Transformations\n",
        "\n",
        "ML.NET supports a variety of data transformations that will convert data into the required format and help you make corrections to your data. Some common operations are manipulating columns, normalizing values, replacing missing values, converting values, and more. \n",
        "\n",
        "For more information, see [data transformations](https://docs.microsoft.com/dotnet/machine-learning/resources/transforms). \n",
        "\n",
        "Below are a few common transformations. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Categorical data\n",
        "\n",
        "One hot encoding is an important transformation for data containing categories. ML algorithms require data to be numerical, it doesn't know how to process strings representing categories. The columns of vendor_id and payment_type are categorical, vendor can be \"CMD\" or \"VST\" and payment can be \"CReDit\" or \"CaSH\". One hot encoding takes the string values passed in and converts them into numerical data."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "dotnet_interactive": {
          "language": "csharp"
        },
        "vscode": {
          "languageId": "csharp"
        }
      },
      "outputs": [],
      "source": [
        "var pipeline = mlContext.Transforms.Categorical.OneHotEncoding(\n",
        "    new[] \n",
        "    { new InputOutputColumnPair(@\"vendor_id\"), \n",
        "    new InputOutputColumnPair(@\"payment_type\")},\n",
        "    OneHotEncodingEstimator.OutputKind.Binary); "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Let's test the above transform on the vendor_id and payment_type. The result should be a vector value for each category. For the case of Vendor_Id, CMT becomes `000` and VST becomes `001`. We'll create a new ModelInputTransformed class for the new converted types. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "dotnet_interactive": {
          "language": "csharp"
        },
        "vscode": {
          "languageId": "csharp"
        }
      },
      "outputs": [],
      "source": [
        "using System.Numerics; \n",
        "\n",
        "public class ModelInputTransformed\n",
        "{\n",
        "    [LoadColumn(0)]\n",
        "    [ColumnName(@\"vendor_id\")]\n",
        "    public VBuffer<Single> Vendor_id { get; set; }\n",
        "\n",
        "    [LoadColumn(1)]\n",
        "    [ColumnName(@\"rate_code\")]\n",
        "    public float Rate_code { get; set; }\n",
        "\n",
        "    [LoadColumn(2)]\n",
        "    [ColumnName(@\"passenger_count\")]\n",
        "    public float Passenger_count { get; set; }\n",
        "\n",
        "    [LoadColumn(3)]\n",
        "    [ColumnName(@\"trip_time_in_secs\")]\n",
        "    public float Trip_time_in_secs { get; set; }\n",
        "\n",
        "    [LoadColumn(4)]\n",
        "    [ColumnName(@\"trip_distance\")]\n",
        "    public float Trip_distance { get; set; }\n",
        "\n",
        "    [LoadColumn(5)]\n",
        "    [ColumnName(@\"payment_type\")]\n",
        "    public VBuffer<Single> Payment_type { get; set; }\n",
        "\n",
        "    [LoadColumn(6)]\n",
        "    [ColumnName(@\"fare_amount\")]\n",
        "    public float Fare_amount { get; set; }\n",
        "}"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "dotnet_interactive": {
          "language": "csharp"
        },
        "vscode": {
          "languageId": "csharp"
        }
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "0 0 1\t\t0 0 1\r\n"
            ]
          },
          "execution_count": 1,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "// Run the transform\n",
        "IDataView transformedData = pipeline.Fit(data).Transform(data);\n",
        "var convertedData = mlContext.Data.CreateEnumerable<ModelInputTransformed>(transformedData, true);\n",
        "\n",
        "// One Hot Encoding of two columns 'vendor_id' and 'payment_type'.\n",
        "Console.WriteLine(\"Vendor_Id\" +\"\\t\" + \"Payment_Type\"); \n",
        "foreach (ModelInputTransformed item in convertedData)\n",
        "{    \n",
        "    Console.WriteLine(\"{0}\\t\\t{1}\", string.Join(\" \", item.Vendor_id.DenseValues()),\n",
        "                    string.Join(\" \", item.Payment_type.DenseValues()));\n",
        "}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Replace missing values \n",
        "\n",
        "Another common operation is to replace missing values. Here we use the default replacement mode, which replaces the value with the default value for its type."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "dotnet_interactive": {
          "language": "csharp"
        },
        "vscode": {
          "languageId": "csharp"
        }
      },
      "outputs": [],
      "source": [
        "pipeline.Append(mlContext.Transforms.ReplaceMissingValues(\n",
        "    new[] { new InputOutputColumnPair(@\"rate_code\", @\"rate_code\"), \n",
        "    new InputOutputColumnPair(@\"passenger_count\", @\"passenger_count\"), \n",
        "    new InputOutputColumnPair(@\"trip_time_in_secs\", @\"trip_time_in_secs\"), \n",
        "    new InputOutputColumnPair(@\"trip_distance\", @\"trip_distance\") })); "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Again, let's run the transform and take a look at the filled in value. We were missing the rate_code for the second dummy object. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "dotnet_interactive": {
          "language": "csharp"
        },
        "vscode": {
          "languageId": "csharp"
        }
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "Rate_code: 0"
            ]
          },
          "execution_count": 1,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "IDataView transformedData = pipeline.Fit(data).Transform(data);\n",
        "var convertedData = mlContext.Data.CreateEnumerable<ModelInputTransformed>(transformedData, true);\n",
        "\n",
        "\"Rate_code: \" + convertedData.ElementAt(1).Rate_code"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Now let's concatenate all of our feature columns into one vector column. Many ML trainers expect features to be of vector type because applying operations to a vector is more efficient."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "dotnet_interactive": {
          "language": "csharp"
        },
        "vscode": {
          "languageId": "csharp"
        }
      },
      "outputs": [],
      "source": [
        "pipeline.Append(mlContext.Transforms.Concatenate(\n",
        "    @\"Features\", new[] { @\"vendor_id\", @\"payment_type\", @\"rate_code\", @\"passenger_count\", @\"trip_time_in_secs\", @\"trip_distance\" }));"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "We now have a loaded IDataView and pipeline to use for training. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Continue learning\n",
        "\n",
        "> [⏩ Next Module - Training and AutoML](./03-Training%20and%20AutoML.ipynb)  \n",
        "> [⏪ Last Module - Intro to Machine Learning](./01-Intro%20to%20Machine%20Learning.ipynb)  "
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": ".NET (C#)",
      "language": "C#",
      "name": ".net-csharp"
    },
    "language_info": {
      "file_extension": ".cs",
      "mimetype": "text/x-csharp",
      "name": "C#",
      "pygments_lexer": "csharp",
      "version": "8.0"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 4
}
